fix(github): create domain accounts for non-committer authors (#8886) - #8894

…#8886) (apache#8894)

* fix(github): create domain accounts for non-committer authors (apache#8886)

ConvertAccounts sourced the domain `accounts` table FROM _tool_github_accounts,
which is only populated for users we collected full profiles for (effectively,
committers). Issue and PR authors who never committed were written into
_tool_github_repo_accounts but never converted, so issues.creator_id and
pull_requests.author_id pointed at accounts rows that didn't exist.

Source ConvertAccounts FROM _tool_github_repo_accounts LEFT JOIN
_tool_github_accounts instead, so every user the repo references gets a domain
account, enriched with profile detail when we have it and login-only otherwise.
The domain id uses the same generator the issue/PR convertors use, so the FKs
line up. Also emit a repo_account for a PR's merged_by user so
pull_requests.merged_by_id resolves too.

The query stays MySQL/PostgreSQL-agnostic (COALESCE, no backtick quoting,
parameterized via the dal) and mirrors the join already in
account_org_collector.go.

Adds the non-committer orphan case to the e2e fixture plus a referential-
integrity assertion in TestAccountDataFlow. Verified on both MySQL and
PostgreSQL.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(github): carry _raw_data provenance and guard zero account ids

Review feedback on apache#8894: the rewritten ConvertAccounts dropped the
_raw_data fields from converted accounts. Select them with COALESCE,
preferring the enriched _tool_github_accounts row and falling back to
the _tool_github_repo_accounts row for non-committers. The e2e fixture
now carries the _raw_data columns like every other tool fixture, and
the regenerated snapshot pins the pre-existing provenance for enriched
accounts plus the issue-extractor provenance for the non-committer
case.

Also guard PR author and merged-by id generation against zero ids: an
unmerged PR or a deleted user otherwise yields the domain id
github:GithubAccount:<conn>:0, the same orphan-FK shape this PR fixes.
issue_convertor already guards its AuthorId the same way.

Verified with the full github e2e suite on both MySQL and PostgreSQL.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Both were already fixed upstream in apache#8894 (fixes apache#8886)
but this fork had drifted from it:

- pr_convertor.go: guard AuthorId/MergedById against 0 before
  generating a domain account ID. Without it, a null author/merged_by
  from the REST API produces a phantom github:GithubAccount:X:0 ID
  that matches no accounts row. Confirmed byte-identical to upstream's
  current code.
- pr_extractor.go: restore the merged_by repo_account emission this
  fork had dropped, so pull_requests.merged_by_id resolves to a real
  account. Confirmed byte-identical to upstream's current code.

DPROD-1342

Co-Authored-By: Claude <noreply@anthropic.com>

…populate IsBot

Adopts apache#8894's unified ConvertAccounts query (FROM
_tool_github_repo_accounts LEFT JOIN _tool_github_accounts) instead of
maintaining this fork's own two-pass design (ConvertAccounts +
convertOrphanedRepoAccounts, added by cd99281 for the same problem).
Upstream's version is more complete (correct _raw_data provenance,
which the two-pass patch had broken) and removes a large source of
rebase conflict risk.

IsBot is populated via isBotAccount() (API Type=="Bot" or well-known
login conventions: [bot]/-bot/-robot suffixes, copilot/dependabot/
github-actions/codecov-commenter logins) plus hasNoProfileData() — an
account with no avatar_url ever collected (real GitHub users always
get one) is almost always a bot that login/type patterns missed. This
replaces the old orphan-pass's blanket "orphans are always bots"
heuristic with an equivalent signal inside the single query.

Validated against live data: closes the exact gap DPROD-1262
described (konflux-ci-triage[bot] and konflux-ci-review[bot], both
previously missing from accounts entirely, now resolve with
IsBot=true) and confirmed on a fresh konflux-ci/devlake collection
(codecov-commenter correctly flagged despite the GitHub API reporting
it as Type: "User" with a full profile).

DPROD-1262
DPROD-1342

Co-Authored-By: Claude <noreply@anthropic.com>

Materializes docs/research/bot-commit-identification.md into the
working tree (previously only existed on the unmerged PR konflux-ci#118 branch)
and revises it to reflect adopting apache#8894 instead of
extending this fork's own two-pass account_convertor design, plus the
codecov-commenter finding from live testing.

Also backfills docs/upstream-diffs.md with the account_convertor.go
divergence this fork already carried (introduced by cd99281,
DPROD-1259, never logged) alongside the new IsBot-specific entries.
pr_convertor.go and pr_extractor.go now match upstream exactly and
need no entry going forward.

DPROD-1342

Co-Authored-By: Claude <noreply@anthropic.com>

Both were already fixed upstream in apache#8894 (fixes apache#8886)
but this fork had drifted from it:

- pr_convertor.go: guard AuthorId/MergedById against 0 before
  generating a domain account ID. Without it, a null author/merged_by
  from the REST API produces a phantom github:GithubAccount:X:0 ID
  that matches no accounts row. Confirmed byte-identical to upstream's
  current code.
- pr_extractor.go: restore the merged_by repo_account emission this
  fork had dropped, so pull_requests.merged_by_id resolves to a real
  account. Confirmed byte-identical to upstream's current code.

DPROD-1342

Co-Authored-By: Claude <noreply@anthropic.com>

…populate IsBot

Adopts apache#8894's unified ConvertAccounts query (FROM
_tool_github_repo_accounts LEFT JOIN _tool_github_accounts) instead of
maintaining this fork's own two-pass design (ConvertAccounts +
convertOrphanedRepoAccounts, added by cd99281 for the same problem).
Upstream's version is more complete (correct _raw_data provenance,
which the two-pass patch had broken) and removes a large source of
rebase conflict risk.

IsBot is populated via isBotAccount() (API Type=="Bot" or well-known
login conventions: [bot]/-bot/-robot suffixes, copilot/dependabot/
github-actions/codecov-commenter logins) plus hasNoProfileData() — an
account with no avatar_url ever collected (real GitHub users always
get one) is almost always a bot that login/type patterns missed. This
replaces the old orphan-pass's blanket "orphans are always bots"
heuristic with an equivalent signal inside the single query.

Validated against live data: closes the exact gap DPROD-1262
described (konflux-ci-triage[bot] and konflux-ci-review[bot], both
previously missing from accounts entirely, now resolve with
IsBot=true) and confirmed on a fresh konflux-ci/devlake collection
(codecov-commenter correctly flagged despite the GitHub API reporting
it as Type: "User" with a full profile).

DPROD-1262
DPROD-1342

Co-Authored-By: Claude <noreply@anthropic.com>

Materializes docs/research/bot-commit-identification.md into the
working tree (previously only existed on the unmerged PR konflux-ci#118 branch)
and revises it to reflect adopting apache#8894 instead of
extending this fork's own two-pass account_convertor design, plus the
codecov-commenter finding from live testing.

Also backfills docs/upstream-diffs.md with the account_convertor.go
divergence this fork already carried (introduced by cd99281,
DPROD-1259, never logged) alongside the new IsBot-specific entries.
pr_convertor.go and pr_extractor.go now match upstream exactly and
need no entry going forward.

DPROD-1342

Co-Authored-By: Claude <noreply@anthropic.com>